Skip to content

Instantly share code, notes, and snippets.

@rishiloyola
Last active February 21, 2022 14:21
Show Gist options
  • Save rishiloyola/79f869749bf54d135f7f6fe61e0e99a7 to your computer and use it in GitHub Desktop.
Save rishiloyola/79f869749bf54d135f7f6fe61e0e99a7 to your computer and use it in GitHub Desktop.
[ELK Stack] Generate TLS certs for filebeat and logstash
1. Generate new domain name for logstash server.
For this tutorial
domain name = logstash-prod.xyz.com
ip = 1.2.3.4
* Enter to following directory
```
$ sudo mkdir /etc/pki
$ cd /etc/pki
```
* Generate CA and self-sign it.
```
$ mkdir -p certs/{devices,client,ca,tmp}
$ openssl genrsa -out certs/ca/root-ca.key.pem 2048
$ openssl req -x509 -new -nodes -key certs/ca/root-ca.key.pem -days 9131 -out certs/ca/root-ca.crt.pem -subj "/C=US/ST=Utah/L=Provo/O=ACME Signing Authority Inc/CN=logstash-prod.xyz.com"
```
* Generate logstash certs
```
$ openssl genrsa -out certs/devices/logstash.key.pem 2048
$ openssl req -new -key certs/devices/logstash.key.pem -out certs/tmp/logstash.csr.pem -subj "/C=US/ST=Utah/L=Provo/O=ACME Service/CN=logstash-prod.xyz.com"
$ openssl x509 -req -in certs/tmp/logstash.csr.pem -CA certs/ca/root-ca.crt.pem -CAkey certs/ca/root-ca.key.pem -CAcreateserial -out certs/devices/logstash.crt.pem -days 9131
```
* Generate filebeat certs
```
$ openssl genrsa -out certs/devices/filebeat.key.pem 2048
$ openssl req -new -key certs/devices/filebeat.key.pem -out certs/tmp/filebeat.csr.pem -subj "/C=US/ST=Utah/L=Provo/O=ACME Service/CN=logstash-prod.xyz.com"
$ openssl x509 -req -in certs/tmp/filebeat.csr.pem -CA certs/ca/root-ca.crt.pem -CAkey certs/ca/root-ca.key.pem -CAcreateserial -out certs/devices/filebeat.crt.pem -days 9131
```
* convert private key to PKCS8 format
```
$ openssl pkcs8 -topk8 -inform pem -in certs/devices/logstash.key.pem -outform pem -nocrypt -out certs/devices/logstash-pkcs8.pem
$ openssl pkcs8 -topk8 -inform pem -in certs/devices/filebeat.key.pem -outform pem -nocrypt -out certs/devices/filebeat-pkcs8.pem
```
* Give `777` file permission to all these certs
* Restart logstash if you did it after starting logstash
`$ cd /etc/deploy/docker-compose && sudo docker-compose down`
* verify it
```
$ curl -v --key certs/devices/filebeat-pkcs8.pem --cert certs/devices/filebeat.crt.pem --cacert certs/ca/root-ca.crt.pem https://logstash-prod.xyz.com:5044
```
@sandipdivekar1
Copy link

Hello Rishi,
I am also working on generating certs for logstash and filebeat communication. I followed your steps but I facing 1 issue.
error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
I have deployed logstash in docker swarm container and running filebeat on multiple ec2 instances. Could you please help, I have tried so many different ways.

I have configured logstash and filebeat as:
logstash.conf:
input {
beats {
client_inactivity_timeout => 3000
port => 5044
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt.pem"
ssl_key => "/etc/logstash/logstash-pkcs8.pem"
ssl_certificate_authorities => ["/etc/logstash/root-ca.crt.pem"]
ssl_verify_mode => "peer"
}
}

filebeat.yml
output.logstash:
hosts: ["54.221.XX.XX:5044"]
ssl.certificate_authorities: ["/root/root-ca.crt.pem"]
ssl.certificate: "/root/filebeat.crt.pem"
ssl.key: "/root/filebeat-pkcs8.pem"

@rishiloyola
Copy link
Author

This is my logstash.conf

input {
	beats {
    		port => 5044
    		ssl => true
            ssl_certificate_authorities => ["/etc/pki/certs/ca/root-ca.crt.pem"]
            ssl_certificate => "/etc/pki/certs/devices/logstash.crt.pem"
            ssl_key => "/etc/pki/certs/devices/logstash-pkcs8.pem"
            ssl_verify_mode => "force_peer"
  	}
}

filter {
  if [fileset][module] == "system" {
    if [fileset][name] == "auth" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][user][add][name]}, UID=%{NUMBER:[system][auth][user][add][uid]}, GID=%{NUMBER:[system][auth][user][add][gid]}, home=%{DATA:[system][auth][user][add][home]}, shell=%{DATA:[system][auth][user][add][shell]}$",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
        pattern_definitions => {
          "GREEDYMULTILINE"=> "(.|\n)*"
        }
        remove_field => "message"
      }
      date {
        match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
      geoip {
        source => "[system][auth][ssh][ip]"
        target => "[system][auth][ssh][geoip]"
      }
    }
    else if [fileset][name] == "syslog" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
        pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
        remove_field => "message"
      }
      date {
        match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
    }
  }
}

output {
	elasticsearch {
		hosts => "127.0.0.1:9200"
		sniffing => true
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
	}
}

@rishiloyola
Copy link
Author

Filebeat conf

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/auth.log

filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  hosts: ["$1:5044"]
  ssl.certificate_authorities: ["/etc/pki/certs/ca/root-ca.crt.pem"]
  ssl.certificate: "/etc/pki/certs/devices/filebeat.crt.pem"
  ssl.key: "/etc/pki/certs/devices/filebeat-pkcs8.pem"

@mukuldeepfence
Copy link

i have followed the steps for generating the Certs

My logstash versiion is 7.10.2
filebeat version is 7.10.2

filebeat.yml

output.logstash:
hosts: ["deepfence-logstash:8005"]
proxy_url: socks5://${DEEPFENCE_KEY}:@${DF_BACKEND_IP}:8005
proxy_use_local_resolver: false
ssl.enabled: true
ssl.certificate_authorities: ["/etc/filebeat/root-ca.crt.pem"]
ssl.certificate: "/etc/filebeat/filebeat.crt.pem"
ssl.key: "/etc/filebeat/filebeat-pkcs8.pem"

input {
beats {
port => "${TCP_PORT}"
codec => json
ssl => true # enable TLS/SSL
ssl_certificate_authorities => ["/etc/logstash/root-ca.crt.pem"]
ssl_certificate => "/etc/logstash/logstash.crt.pem"
ssl_key => "/etc/logstash/logstash-pkcs8.pem"
ssl_verify_mode => "force_peer"
client_inactivity_timeout => 600
}
}

and i am getting this error -
[DEBUG] 2021-04-20 16:16:44.748 [defaultEventExecutorGroup-4-2] BeatsHandler - [local: 0.0.0.0:8005, remote: 192.168.128.15:58986] Handling exception: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain (caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain)
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain

@mukuldeepfence
Copy link

My problem has been solved i created the wrong certs by adding the port in the domain name.
when i removed the port name from the domain name then all worked as of now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment