ableasdale/mtls-kafka.md

## mtls-kafka.md

      
    Raw
  

              mtls-kafka.md
            
          
    Kafka and mTLS Walkthrough

Creating a Certificate Authority (CA)

Build the Certificate

openssl req -new -newkey rsa:4096 -days 365 -x509 -subj "/CN=Kafka-Security-CA" -keyout ca-key -out ca-cert -nodes
You should see:
Generating a RSA private key
...................................................++++
..............................................................................................................................................................++++
writing new private key to 'ca-key'
And the directory will now contain
-rw-rw-r-- 1 ubuntu ubuntu 1830 Apr  7 21:58 ca-cert
-rw------- 1 ubuntu ubuntu 3272 Apr  7 21:58 ca-key
ca-key is the private key:
-----BEGIN PRIVATE KEY-----
ca-cert is the public certificate that can be imported into the trust store later:
-----BEGIN CERTIFICATE-----
You can now use this CA when you secure your Kafka broker.
Create the Keystore

keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=host-dns-name.compute.amazonaws.com" -storetype pkcs12 -keyalg RSA
Where -dname is the public DNS name of the host
You will now see:
-rw-rw-r-- 1 ubuntu ubuntu 1830 Apr  7 21:58 ca-cert
-rw------- 1 ubuntu ubuntu 3272 Apr  7 21:58 ca-key
-rw-rw-r-- 1 ubuntu ubuntu 2359 Apr  7 22:22 kafka.server.keystore.jks
Check the jks file:
keytool -list -v -keystore kafka.server.keystore.jks
You'll see something like:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry
Look at the Owner and make sure the CN matches:
Owner: CN=host-dns-name.compute.amazonaws.com
Create the cert file (a signed version of the certificate)

This step creates an output file called cert-file which the brokers can use.  This step is the first part of a 2-step process and is commonly called a signing request.
keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass confluent -keypass confluent
You should now see a cert-file has been generated in the directory:
-rw-rw-r-- 1 ubuntu ubuntu 1509 Apr  7 22:42 cert-file
The cert-file that has just been output would normally be sent over to the CA Administrator to get the certificate signed.  In our case, we're just going to self-sign the certificate.
Self Sign the Certificate

The most important switches are:

req : a request
CA : the Certificate Authority
CAKey : the public key
in : the input (the signing request)
out : the signed certificate

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:confluent
You should see:
Signature ok
subject=CN = host-dns-name.compute.amazonaws.com
Getting CA Private Key
And your directory should now contain a cert-signed file:
-rw-rw-r-- 1 ubuntu ubuntu 2139 Apr  8 07:53 cert-signed
Examine the signed certificate file by running:
keytool -printcert -v -file cert-signed
Where:

printcert : request to print the certificate
v : uses verbose mode

You should see something like this in the output:
Owner: CN=host-dns-name.compute.amazonaws.com
Issuer: CN=Kafka-Security-CA
Serial number: 4a3dfd85781b74852c77194b7665b0b31870c75e
Valid from: Fri Apr 08 07:53:18 UTC 2022 until: Sat Apr 08 07:53:18 UTC 2023
Certificate fingerprints:
	 SHA1: 73:87:AC:60:B7:2F:6E:14:A8:01:D6:0C:63:5D:59:CB:B9:B1:B4:AB
	 SHA256: A2:C9:A2:A3:1A:EA:FC:CF:54:B4:84:1A:11:06:73:29:26:18:26:CF:42:1A:3D:42:2C:18:1B:5E:C9:81:C5:5D
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit DSA key
Version: 1
Note the Owner and the Issuer fields in the output.
Create the Trust Store

The next part of the process is to create our truststore on the Kafka Broker:
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA

keystore : the jks truststore
alias : ???
import : import our own CA public certificate

You should see:
Certificate was added to keystore
and in your directory listing, you should now see a new file (kafka.server.truststore.jks):
-rw-rw-r-- 1 ubuntu ubuntu 1671 Apr  8 09:30 kafka.server.truststore.jks
Import the certificate into the trust store

For the next stage, we need to import the new certificates into our keystore.  Now we have a signed certificate and a ca-certificate; both of these need to be stored in the keystore.
First import the public ca-cert certificate into the keystore:
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt
At this stage, all the switches should look familiar...
You should see the message:
Certificate was added to keystore
Then import the cert-signed signed certificate into the keystore:
keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass confluent -keypass confluent -noprompt
You should see:
Certificate reply was installed in keystore
Configure the broker

vim kafka/config/server.properties
You'll be adding the following lines:
# TLS / SSL
ssl.keystore.location=/home/ubuntu/ssl/kafka.server.keystore.jks
ssl.keystore.password=confluent
ssl.key.password=confluent
ssl.truststore.location=/home/ubuntu/ssl/kafka.server.truststore.jks
ssl.truststore.password=confluent
You also need to ensure you have both listeners registered (the PLAINTEXT listener on port 9092 and the SSL listener on port 9093).  You need to configure the listeners property in order for this mapping to be created in the broker.
You also need to extend advertised.listeners to add the new SSL listener; note that we're specifying the SSL port for your SSL connection in the advertised.listeners property:
listeners=PLAINTEXT://:9092,SSL://:9093
advertised.listeners=PLAINTEXT://host-dns-name.compute.amazonaws.com:9092,SSL://host-dns-name.compute.amazonaws.com:9093
Now we need to restart Kafka:
sudo systemctl restart kafka && sudo systemctl status kafka
We want to see that the broker is in a running state after the configuration changes have been made:
● kafka.service - Apache Kafka server (broker)
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-04-08 12:52:32 UTC; 14ms ago
Verify connectivity by running:
openssl s_client -connect host-dns-name.compute.amazonaws.com:9093 -cipher ALL
You should now see something like:
CONNECTED(00000003)
depth=1 CN = Kafka-Security-CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 CN = Kafka-Security-CA
verify return:1
depth=0 CN = host-dns-name.compute.amazonaws.com
verify return:1
---
Grep for:
grep "endpoint" kafka/logs/server.log
You should see the SSL listener reported as "Started":
[2022-04-09 08:41:37,820] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane acceptor and processor(s) for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer)
[2022-04-09 08:41:37,849] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane acceptor and processor(s) for endpoint : ListenerName(SSL) (kafka.network.SocketServer)

Check the key

keytool -keystore ssl/kafka.client.keystore.jks -list -v
Note that Criticality=false in some cases.
BasicConstraints CA:true (self signed)
See: https://github.com/confluentinc/cp-demo/blob/7.1.0-post/scripts/security/certs-create-per-user.sh#L43
certs-create-per-user.sh
Putting it all together

All the commands back-to-back:
keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=host-dns-name.compute.amazonaws.com" -storetype pkcs12 -keyalg RSA
keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass confluent -keypass confluent
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:confluent
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt
keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass confluent -keypass confluent -noprompt
sudo systemctl restart kafka && sudo systemctl status kafka
debugging notes


openssl s_client -connect host-dns-name.compute.amazonaws.com:9093 -CAfile ssl/ca-cert -tls1_2
openssl s_client -debug -connect host-dns-name.compute.amazonaws.com:9093

Securing the client

Prerequisites

First ensure that you have the ca-cert file (the public CA certificate) on your machine - in our case, we created this certificate on our remote Kafka broker machine, so we will need to scp it over:
scp -i ~/key.pem ubuntu@host-dns-name.compute.amazonaws.com:~/ssl/ca-cert .
Check the directory on the machine that will be used for the client application for the ca-cert file:
-rw-r--r--  1 ableasdale  staff   1.8K 10 Apr 09:01 ca-cert
Create the Client Trust Store


ca-cert is the public CA certificate
alias is the CA Certificate alias name - we're using the same CARoot alias that was used previously

This will create your truststore file
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA
You should see the message:
Certificate was added to keystore
Check the keystore file:
keytool -list -v -keystore kafka.client.truststore.jks
You will be prompted for the password and then you should see something like this:
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry
Note the Alias name and the Owner:
Alias name: caroot
[...]
Owner: CN=Kafka-Security-CA
Issuer: CN=Kafka-Security-CA
Create the Client Certificate

This will create your keystore file (kafka.client.keystore.jks):
keytool -genkey -keystore kafka.client.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=mylaptop" -alias my-local-pc -storetype pkcs12 -keyalg RSA
Create Certificate Signing Request

This will create the client-cert-sign-request file for your client application:
keytool -keystore kafka.client.keystore.jks -certreq -file client-cert-sign-request -alias my-local-pc -storepass confluent -keypass confluent
In a production situation, this signing request would be sent to your Certificate Admin for signing.  However, as before, we're going to rely on self-signing.
SCP the file to our broker machine

scp -i ~/key.pem client-cert-sign-request ubuntu@host-dns-name.compute.amazonaws.com:~/ssl
You should now see the client-cert-sign-request in the ssl directory on the broker:
-rw-r--r-- 1 ubuntu ubuntu  977 Apr 10 07:19 client-cert-sign-request
SSH into our EC2 instance

ssh -i key.pem ubuntu@host-dns-name.compute.amazonaws.com
cd to the ssl directory
Sign the Certificate

Remember:

ca-cert is the public CA certificate
ca-key is the private key for the CA certificate
in is the input file (client-certificate-sign-request)
out is the output file (client-cert-signed)

As with the broker section, we will use the openssl commandline tool to sign the client certificate:
openssl x509 -req -CA ca-cert -CAkey ca-key -in client-cert-sign-request -out client-cert-signed -days 365 -CAcreateserial -passin pass:confluent
You should see:
Signature ok
subject=CN = mylaptop
Getting CA Private Key
Your directory listing will now contain the new client-cert-signed certificate:
-rw-rw-r-- 1 ubuntu ubuntu 1346 Apr 10 07:29 client-cert-signed
Copy the signed client certificate back to the client machine

We're going to use scp to copy the client certificate back to the machine running the client applications (in this case, it's a laptop):
From your client machine (i.e. your laptop), run:
scp -i ~/key.pem ubuntu@host-dns-name.compute.amazonaws.com:~/ssl/client-cert-signed .
You'll now see this in the directory:
-rw-r--r--  1 ableasdale  staff   1.3K 10 Apr 08:55 client-cert-signed
Import the signed client certificate and the CA into the keystore

Now you can add the public CA certificate into the client keystore.  We SCPd this file over earlier in the process.
Notes:

keystore is the client keystore file (we're adding the root certificate to the store)
alias this is the CA Certificate so we're using the same CARoot alias that was used previously
file is the public CA certificate

keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt
You should see:
Certificate was added to keystore
Now we add the client signed certificate (client-cert-signed) using the keytool command:
keytool -keystore kafka.client.keystore.jks -import -file client-cert-signed -alias my-local-pc -storepass confluent -keypass confluent -noprompt
You should see:
Certificate reply was installed in keystore
Configure the Kafka Broker to require SSL client Authentication

Now we need to edit the server.properties on the broker:
vim ~/kafka/config/server.properties
We need to add the following line to the broker's properties file:
# Client Authentication
ssl.client.auth=required
Restart Kafka to apply the new configuration changes:
sudo systemctl restart kafka && sudo systemctl status kafka
Create the Client SSL Authentication Properties File

On your client machine, you need to create a properties file:
vim client-ssl-auth.properties
We're adding the following lines to the client-ssl-auth.properties properties file:
security.protocol=SSL
ssl.truststore.location=~/ssl/kafka.client.truststore.jks
ssl.truststore.password=confluent
ssl.keystore.location=~/ssl/kafka.client.keystore.jks
ssl.keystore.password=confluent
ssl.key.password=confluent
Test the Producer

We're using kafka-console-producer.sh in the bin directory of Apache Kafka and we're passing in the client-ssl-auth.properties that we created in the previous step:
~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --broker-list host-dns-name.compute.amazonaws.com:9093 --topic kafka-security-topic --producer.config client-ssl-auth.properties
Test the Consumer

We're using kafka-console-consumer.sh in the bin directory of Apache Kafka and we're passing in the client-ssl-auth.properties that we created in the previous step:
~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server host-dns-name.compute.amazonaws.com:9093 --topic kafka-security-topic --consumer.config client-ssl-auth.properties
Non-TLS Producer and Consumer

Note that we have configured our broker with two listeners; there's the plaintext listener (on port 9092) and the SSL listener (on port 9093).  We can still use the non-TLS endpoint to communicate with the brokers:
Producer

~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --broker-list host-dns-name.compute.amazonaws.com:9092 --topic kafka-security-topic
Consumer

~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server host-dns-name.compute.amazonaws.com:9092 --topic kafka-security-topic