[ELK Stack] Generate TLS certs for filebeat and logstash

gistfile1.txt

1. Generate new domain name for logstash server.

For this tutorial 
domain name = logstash-prod.xyz.com
ip = 1.2.3.4

* Enter to following directory
```
$ sudo mkdir /etc/pki
$ cd /etc/pki
```

* Generate CA and self-sign it.

```
$ mkdir -p certs/{devices,client,ca,tmp}
$ openssl genrsa -out certs/ca/root-ca.key.pem 2048
$ openssl req -x509 -new -nodes -key certs/ca/root-ca.key.pem -days 9131 -out certs/ca/root-ca.crt.pem -subj "/C=US/ST=Utah/L=Provo/O=ACME Signing Authority Inc/CN=logstash-prod.xyz.com"
```

* Generate logstash certs

```
$ openssl genrsa -out certs/devices/logstash.key.pem 2048
$ openssl req -new -key certs/devices/logstash.key.pem -out certs/tmp/logstash.csr.pem -subj "/C=US/ST=Utah/L=Provo/O=ACME Service/CN=logstash-prod.xyz.com"
$ openssl x509 -req -in certs/tmp/logstash.csr.pem -CA certs/ca/root-ca.crt.pem -CAkey certs/ca/root-ca.key.pem -CAcreateserial -out certs/devices/logstash.crt.pem -days 9131
```

* Generate filebeat certs
```
$ openssl genrsa -out certs/devices/filebeat.key.pem 2048
$ openssl req -new -key certs/devices/filebeat.key.pem -out certs/tmp/filebeat.csr.pem -subj "/C=US/ST=Utah/L=Provo/O=ACME Service/CN=logstash-prod.xyz.com"
$ openssl x509 -req -in certs/tmp/filebeat.csr.pem -CA certs/ca/root-ca.crt.pem -CAkey certs/ca/root-ca.key.pem -CAcreateserial -out certs/devices/filebeat.crt.pem -days 9131
```

* convert private key to PKCS8 format
```
$ openssl pkcs8 -topk8 -inform pem -in certs/devices/logstash.key.pem -outform pem -nocrypt -out certs/devices/logstash-pkcs8.pem
$ openssl pkcs8 -topk8 -inform pem -in certs/devices/filebeat.key.pem -outform pem -nocrypt -out certs/devices/filebeat-pkcs8.pem
```
* Give `777` file permission to all these certs

* Restart logstash if you did it after starting logstash
`$ cd /etc/deploy/docker-compose && sudo docker-compose down`

* verify it
```
$ curl -v --key certs/devices/filebeat-pkcs8.pem --cert certs/devices/filebeat.crt.pem --cacert certs/ca/root-ca.crt.pem https://logstash-prod.xyz.com:5044
```

Hello Rishi,
I am also working on generating certs for logstash and filebeat communication. I followed your steps but I facing 1 issue.
error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
I have deployed logstash in docker swarm container and running filebeat on multiple ec2 instances. Could you please help, I have tried so many different ways.

I have configured logstash and filebeat as:
logstash.conf:
input {
beats {
client_inactivity_timeout => 3000
port => 5044
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt.pem"
ssl_key => "/etc/logstash/logstash-pkcs8.pem"
ssl_certificate_authorities => ["/etc/logstash/root-ca.crt.pem"]
ssl_verify_mode => "peer"
}
}

filebeat.yml
output.logstash:
hosts: ["54.221.XX.XX:5044"]
ssl.certificate_authorities: ["/root/root-ca.crt.pem"]
ssl.certificate: "/root/filebeat.crt.pem"
ssl.key: "/root/filebeat-pkcs8.pem"

This is my logstash.conf

input {
	beats {
    		port => 5044
    		ssl => true
            ssl_certificate_authorities => ["/etc/pki/certs/ca/root-ca.crt.pem"]
            ssl_certificate => "/etc/pki/certs/devices/logstash.crt.pem"
            ssl_key => "/etc/pki/certs/devices/logstash-pkcs8.pem"
            ssl_verify_mode => "force_peer"
  	}
}

filter {
  if [fileset][module] == "system" {
    if [fileset][name] == "auth" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
        pattern_definitions => {
          "GREEDYMULTILINE"=> "(.|\n)*"
        }
        remove_field => "message"
      }
      date {
        match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
      geoip {
        source => "[system][auth][ssh][ip]"
        target => "[system][auth][ssh][geoip]"
      }
    }
    else if [fileset][name] == "syslog" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
        pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
        remove_field => "message"
      }
      date {
        match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
    }
  }
}

output {
	elasticsearch {
		hosts => "127.0.0.1:9200"
		sniffing => true
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
	}
}

Filebeat conf

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/auth.log

filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  hosts: ["$1:5044"]
  ssl.certificate_authorities: ["/etc/pki/certs/ca/root-ca.crt.pem"]
  ssl.certificate: "/etc/pki/certs/devices/filebeat.crt.pem"
  ssl.key: "/etc/pki/certs/devices/filebeat-pkcs8.pem"

i have followed the steps for generating the Certs

My logstash versiion is 7.10.2
filebeat version is 7.10.2

filebeat.yml

output.logstash:
hosts: ["deepfence-logstash:8005"]
proxy_url: socks5://${DEEPFENCE_KEY}:@${DF_BACKEND_IP}:8005
proxy_use_local_resolver: false
ssl.enabled: true
ssl.certificate_authorities: ["/etc/filebeat/root-ca.crt.pem"]
ssl.certificate: "/etc/filebeat/filebeat.crt.pem"
ssl.key: "/etc/filebeat/filebeat-pkcs8.pem"

input {
beats {
port => "${TCP_PORT}"
codec => json
ssl => true # enable TLS/SSL
ssl_certificate_authorities => ["/etc/logstash/root-ca.crt.pem"]
ssl_certificate => "/etc/logstash/logstash.crt.pem"
ssl_key => "/etc/logstash/logstash-pkcs8.pem"
ssl_verify_mode => "force_peer"
client_inactivity_timeout => 600
}
}

and i am getting this error -
[DEBUG] 2021-04-20 16:16:44.748 [defaultEventExecutorGroup-4-2] BeatsHandler - [local: 0.0.0.0:8005, remote: 192.168.128.15:58986] Handling exception: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain (caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain)
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

My problem has been solved i created the wrong certs by adding the port in the domain name.
when i removed the port name from the domain name then all worked as of now