The purpose of this document is to help with configuring and troubleshooting using TLS on the connection between Beats and Logstash.
You must configure TLS on both the client and server to make this work. This configuration will ensure that Beats do not leak data by sending data to an untrusted Logstash server. This configuration does not provide client authentication (for that we require logstash-input-beats [issue #8] (logstash-plugins/logstash-input-beats#8) to be fixed).
logstash-input-beats configuration (reference docs):
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/logging/logs.andrewkroh.com.crt"
ssl_key => "/etc/pki/logging/logs.andrewkroh.com.key"
}
}
Beats Logstash output configuration (reference docs):
output:
logstash:
hosts: ["logs.andrewkroh.com:5044"]
tls:
certificate_authorities:
- /etc/pki/logging/ca.crt
You should create a certificate authority (CA) and then sign the server certificate used by Logstash with the CA certificate. Then you configure Beats to trust any certificate signed by your CA certificate.
This document does not address how to create the certificates. There are many online resources describing the process.
NOTE: The protocol between Beats and Logstash is NOT based on HTTP!
You can use curl to validate the Logstash server's certificate even though the
protocol is not based on HTTP. You should receive an empty response error as opposed
to a SSL error.
First let's test the connection using using the IP address in the URL.
curl -v --cacert ca.crt https://192.168.99.100:5044
* Rebuilt URL to: https://192.168.99.100:5044/
* Trying 192.168.99.100...
* Connected to 192.168.99.100 (192.168.99.100) port 5044 (#0)
* WARNING: using IP address, SNI is being disabled by the OS.
* SSL: certificate verification failed (result: 5)
* Closing connection 0
curl: (51) SSL: certificate verification failed (result: 5)
It failed. The reason is because the certificate being used in not valid for
the IP address, it is valid only for the hostname logs.andrewkroh.com. For
work-arounds to this problem see the
troubleshooting section.
Try the test again but use the hostname in the request URL.
$ curl -v --cacert ca.crt https://logs.andrewkroh.com:5044
* Rebuilt URL to: https://logs.andrewkroh.com:5044/
* Trying 192.168.99.100...
* Connected to logs.andrewkroh.com (192.168.99.100) port 5044 (#0)
* TLS 1.2 connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
* Server certificate: logs.andrewkroh.com
* Server certificate: andrewkroh.com
> GET / HTTP/1.1
> Host: logs.andrewkroh.com:5044
> User-Agent: curl/7.43.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host logs.andrewkroh.com left intact
curl: (52) Empty reply from server
It works!
Test your Beat setup by running the Beat in the foreground so you can quickly see any errors that occur. If you have the Beat running as a service, stop it first.
$ <beatname> -c <beatname>.yml -e -v
Any errors will be printed to the console.
This happens because your certificate is only valid for the hostname present in the Subject field.
The possible works-arounds are:
- Create a DNS entry for the hostname mapping it to the server's IP.
- Create an entry in
/etc/hostsfor the hostname. Or on Windows add an entry toC:\Windows\System32\drivers\etc\hosts. - Re-create the server certificate and add a SubjectAltName (SAN) for the IP address of the server. This make the server's certificate valid for both the hostname and the IP address.
OK, so you just want to get up and running fast. Use the insecure: true option
in you beat configuration files. This will cause Beats to ignore any TLS
security issues.
output:
logstash:
hosts: ["logs.andrewkroh.com:5044"]
tls:
insecure: true
2016/02/25 19:56:31.441724 transport.go:125: ERR SSL client failed to connect with: x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs
2016/02/25 19:56:31.441748 single.go:126: INFO Connecting error publishing events (retrying): x509: cannot validate certificate for 192.168.99.100 because it doesn't contain any IP SANs
See Your SSL connection fails when using the IP address of the server.
2016/02/25 17:37:08.496727 transport.go:125: ERR SSL client failed to connect with: dial tcp logs.andrewkroh.com:5044: getsockopt: no route to host
This is not a TLS problem. It's a networking problem. Make sure the two hosts can communicate.
2016/02/25 22:04:06.115207 transport.go:125: ERR SSL client failed to connect with: dial tcp 192.168.99.100:5044: getsockopt: connection refused
This is not a TLS problem. Make sure that Logstash is running and that there is no firewall blocking the traffic.
A firewall is refusing the connection. Check if a firewall is blocking the traffic on the client, the network, or the destination host.
$ openssl x509 -text -noout -in logs.andrewkroh.com.crt
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4 (0x4)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=andrewkroh.com
Validity
Not Before: Feb 25 20:12:53 2016 GMT
Not After : Feb 22 20:12:53 2026 GMT
Subject: CN=logs.andrewkroh.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (2048 bit)
Modulus (2048 bit):
00:bd:b8:0c:2b:d0:09:df:fd:ad:76:ae:53:8f:4c:
47:56:2b:98:5b:bf:98:9a:7e:5d:b9:9a:fb:f0:e8:
62:bc:53:83:ef:90:a9:ba:c5:2e:cb:49:97:89:d7:
45:ae:4a:8d:7b:38:a4:ce:30:02:88:ac:a9:01:06:
38:90:d4:93:45:2d:eb:7d:15:cc:16:04:cf:2e:dc:
a0:a9:e6:11:90:0e:37:d8:12:be:21:8d:2d:74:c3:
b1:da:40:04:a3:57:17:51:8c:d2:ac:2f:53:2a:c4:
78:4a:97:6a:5c:54:79:37:ad:92:8a:a9:69:ac:92:
97:98:e7:68:8d:47:ed:81:4e:0e:7f:f1:0b:25:5e:
4f:92:19:d4:b0:03:77:0e:12:6e:76:59:81:3e:fb:
5c:05:eb:12:b1:63:3a:31:b2:a9:78:fb:c5:08:00:
8f:4f:80:5c:48:d5:0e:8b:d3:11:60:df:99:f4:43:
bb:8b:21:49:f9:2a:3b:84:4c:ad:92:49:17:8b:76:
f0:92:3b:eb:7e:e8:9d:20:1c:65:06:24:a8:87:06:
3f:0b:9e:ec:92:05:60:43:42:10:c2:5e:5a:a1:d7:
95:80:c8:e1:57:2f:8a:e8:54:92:02:fe:38:3c:f6:
ac:46:33:53:c9:48:b5:6e:a2:5c:aa:bc:2c:db:8c:
e1:5f
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
X509v3 Subject Key Identifier:
94:F4:2A:C9:E5:8A:68:78:33:45:5F:1A:E9:F8:19:3B:54:E4:0B:3D
X509v3 Authority Key Identifier:
keyid:03:47:B8:E1:45:97:DD:B7:F4:C3:0B:31:B4:0A:F8:76:6E:99:3B:5B
DirName:/CN=andrewkroh.com
serial:99:4D:A2:8A:02:99:9C:34
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Key Usage:
Digital Signature, Key Encipherment
X509v3 Subject Alternative Name:
IP Address:192.168.99.100
Signature Algorithm: sha256WithRSAEncryption
9d:7d:d3:6f:6b:9b:64:08:d9:21:e9:cc:48:c0:51:7e:a9:84:
6e:8a:b6:4b:c3:63:40:a6:3a:fc:0b:4a:bc:3b:a6:db:3e:2e:
18:b5:35:c3:ab:9f:87:30:ef:ef:46:21:ab:44:fc:bc:b9:93:
62:6a:b8:7c:47:31:2f:4e:82:39:f1:bb:de:58:f6:aa:e0:01:
ff:1f:28:1d:cf:17:55:2a:0d:4a:97:2f:ee:63:97:3d:f5:fe:
4b:11:db:2d:85:9a:0b:04:b2:1e:63:8a:dc:a4:5c:6a:c0:c5:
f6:ec:e6:22:90:ea:fd:a6:a5:9e:29:fc:e1:6d:f2:8f:db:4b:
b7:dc:58:37:6a:fe:c2:e5:25:7b:ae:f6:57:b0:e7:f4:73:c8:
3a:d9:bb:65:5c:e3:e1:53:9c:fd:79:39:88:18:a9:f9:1b:26:
58:8a:55:42:5a:f3:d2:c5:d4:f3:7c:88:57:d0:81:15:40:eb:
31:38:3e:ad:65:ff:5a:1a:c4:c6:42:fc:2a:cc:a2:d8:d3:f8:
a1:33:a2:da:00:9b:71:91:c5:bf:7e:70:53:ea:81:89:51:9f:
4f:04:00:cf:cb:f1:49:bd:4d:6e:8f:4a:2b:57:4c:12:34:3a:
c0:1a:87:79:aa:e3:47:55:cd:d7:ab:fc:06:e9:65:b9:a2:8e:
11:a4:1f:53
The example above shows a certificate that is valid for both
logs.andrewkroh.com and 192.168.99.100.
The hostname is part of the Subject field:
Subject: CN=logs.andrewkroh.com
The IP address is included as a X509 v3 Subject Alternative Name extension field.
X509v3 Subject Alternative Name:
IP Address:192.168.99.100