Using syslog-ng to index events to your [Elasticsearch (ES)][ES] cluster has been possible natively for some time now ([see][1] [various][2] [blog][3] posts), but now with syslog-ng 3.9, you can even use Search Guard to encrypt and authenticate your connections to Elasticsearch.
The main reasons we're using syslog-ng over Logstash (LS) at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3) are:
- superior performance,
- lower system footprint, and
- higher maintainability.
One of the reasons Elasticsearch is a popular choice to store and query logs is the fact that it's an opensource software (OSS) solution. Unfortunately, one of its drawbacks is security: the only official way to use flexible authorization policies and strong cryptography is to use [Shield][shield] (recently renamed to [X-pack][xpack]). And guess what, that's where the free part stops: Shield requires a full ES license, which can quickly empty your pockets, especially if your cluster grows.
Fortunately, the [Search Guard (SG)][SG] project implements strong security in the form of an ES plugin. Its core features are opensource, with additional functionality requiring a license. However, even the latter code is publicly available on GitHub. The license is free for non-commercial use, and much more reasonably priced than Shield for commercial usage: Search Guard is sponsored by its company: floragunn, true to its motto ["Scale your cluster, not your cost"][FGmotto].
That's where the subject of this post comes in: Search Guard support got [merged][syslog_ng-sg] to syslog-ng! Moreover, the Search Guard component required by syslog-ng doesn't require a commercial license, so you can use it for free. Hopefully after reading this post, you'll be able to fly your ESK stack to higher security standards.
This guide targets ES 2.4.1 (official [distribution from elastic.co][ES-release]), and [SG-ssl 2.4.1.16][SG-release] Versions for ES5 are [already available][SG5], and instructions shouldn't change much.
Search Guard started as a free plugin for ES 1.x, then was rewritten completely for ES 2.x, and recently gained support for ES 5.x (yes, elastic.co product's versioning is a total nightmare, and look! floragunn follows rapid change closely). Its main components are Search Guard and Search Guard SSL (both got merged since 5.x to ease deployment). We're not going into much detail about the former, as you'll be able to find a lot of info on its [homepage][SG-SSL] and [documentation][SG-SSL-doc]. Suffice it to say that its function is authentication and authorization, and that it supports many mechanisms like http simple, proxy, kerberos, etc. Also note that its developer team is very responsive, and very efficient at running after the rapid and unfortunate breaking of ES APIs by elastic.co. We'll discuss some more on the latter: Search Guard SSL is the plugin that handles encryption: it's the only one needed by syslog-ng, because it will talk natively with the other ES nodes on the encrypted transport layer (aka the 9300 tcp port).
Citing the [project page][SG-SSL]:
Search Guard (®) SSL is a free and open source security plugin for Elasticsearch which provides SSL/TLS for the transport- and REST-layer.
- Node-to-node encryption through SSL/TLS (Transport layer)
- Secure REST layer through HTTPS (SSL/TLS)
- Supports JDK SSL and Open SSL
- Works with Kibana 4, logstash and beats
We kindly invite the developers to add syslog-ng to that list as soon as this blog post is published ;-)
In order to connect syslog-ng to your Search Guard SSL enabled ES cluster, you will need the following:
- syslog-ng-3.9.1
- a working search-guard-ssl enabled ES cluster
- Elasticsearch installed on the syslog-ng node. (You need the same Elasticsearch version that the cluster uses, but it is not necessary to actually run it)
- a copy of this blog post (or two)
For the impatient, the steps to follow are:
- Install the Search Guard SSL plugin on the syslog-ng node
- Generate a certificate for the syslog-ng node and add it to the cluster nodes' keystore
- Configure syslog-ng's
elasticsearch2destination to useclient-mode(searchguard) - Test
- Enjoy
sudo /usr/share/elasticsearch/bin/plugin install -b com.floragunn/search-guard-ssl/2.4.1.16
Managing SG certificates is beyond the scope of this article. Please refer to the SG-SSL documentation regarding [certificates][SG-SSL-cert] for more details. The logical steps are the following:
- Generate a new key for your syslog-ng node
- Generate a CSR and send it to your CA (or self)
- CA sends back signed certificate
- Add the latter to the keystore of the node (
SYSLOG_NG-NODE_NAME-keystore.jks)
For a quick'n'dirty PoC you can use floragunn's [certificate generator][FGcertgen] to get started in a breeze.
The configuration is pretty straightforward, just follow the [destination's documentation][syslog_ng-elastic-doc] using the correct client-mode.
In the examples, 192.168.42.42 is the syslog-server, and 192.168.42.69 one of the elasticsearch cluster nodes.
The following example will push messages to a daily index with a fixed type:
File: /etc/syslog-ng/syslog-ng.conf
destination d_elasticsearch {
elasticsearch2(
client-lib-dir("/usr/share/elasticsearch/plugins/search-guard-ssl/*.jar:/usr/share/elasticsearch/lib")
index("syslog-${YEAR}.${MONTH}.${DAY}")
type("syslog")
time-zone("UTC")
client_mode("searchguard")
resource("/etc/syslog-ng/elasticsearch.yml")
);
};
File: /etc/syslog-ng/elasticsearch.yml
cluster:
name: elasticsearch
discovery:
zen:
ping:
unicast:
hosts:
- 192.168.42.42
- 192.168.42.69
node:
name: syslog_ng_secure
data; false
master: false
path:
home: /etc/syslog-ng
conf: /etc/syslog-ng
searchguard:
ssl:
transport:
keystore_filepath: syslog_ng-keystore.jks
keystore_password: changeit
truststore_filepath: truststore.jks
truststore_password: changeit
enforce_hostname_verification: trueThe key- and trust-stores should be placed into the directory path.conf = /etc/syslog-ng.
The thorough reader might have frowned at the value of
client-lib-dir(). We invite them to read the contents of github issue/1274 for more information on that matter. The unthorough reader may silently ignore this note.
__--__--__--__--_____
( )
( Elasticsearch Cluster )
( )
( )
( es-node-1 )
( 192.168.42.69 )
+--------------+ ( ↑ )
| syslog-ng | ( | )
|192.168.42.42 | ---(sg)--> ( (sg) )
+--------------+ ( | )
( ↓ )
( es-node-2 )
__ ( 192.168.42.79 )
( o> ( )
///\ ( )
\V_/_ (__---__-__--_--__)
The first thing to do is to test the syntax of our syslog-ng config using syslog-ng -f /etc/syslog-ng/syslog-ng.conf -s.
The easiest way to test the config is then to run syslog-ng in the foreground using the command-line switch -F, and to activate the Full Monty debugging using the switches -d and -v: syslog-ng -f /etc/syslog-ng/syslog-ng.conf -Fdv.
If everything goes as expected, you should see something like the following:
[2016-12-20T13:19:08.289217] Module loaded and initialized successfully; module='mod-java'
[2016-12-20T13:19:08.303180] Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='mod-java.so', module='mod-java'
[2016-12-20T13:19:08.303201] Registering candidate plugin; module='mod-java', context='destination', name='java', preference='0'
[2016-12-20T13:19:08.452898] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/java-modules.jar;
[2016-12-20T13:19:08.453308] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/syslog-ng-core.jar;
[2016-12-20T13:19:08.453466] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/elastic-v2.jar;
[2016-12-20T13:19:08.455703] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/search-guard-ssl-2.4.1.16.jar;
[2016-12-20T13:19:08.738063] [syslog_ng] modules [], plugins [search-guard-ssl], sites [];
[…]
[2016-12-20T13:19:09.997536] Worker thread started; driver='d_elasticsearch#0'
[2016-12-20T13:19:09.997870] Checking cluster state..., cluster_name='elasticsearch';
[2016-12-20T13:19:10.043771] [id: 0xb6975e55, /192.168.42.42:55330 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.052281] Cluster is ready to work, cluster_name='elasticsearch';
[…now that everything's set up logs start to come in…]
[2016-12-20T13:20:34.548044] Incoming log entry; line='<27>Dec 20 13:20:34 root: /etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1'
[…and get written to elasticsearch…]
[2016-12-20T13:20:34.550464] Outgoing message; message='2016-12-20T13:20:34+01:00 root err /etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1\x0a'
[2016-12-20T13:20:34.550878] Outgoing log entry, json='{"PROGRAM":"root","PRIORITY":"err","PID":"11636","MESSAGE":"/etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1","ISODATE":"2016-12-20T12:20:34+00:00","HOST":"ccswissrp","FACILITY":"daemon","@timestamp":"2016-12-20T12:20:34+00:00"}';
[2016-12-20T13:20:35.078198] Start bulk processing, id='1';
[2016-12-20T13:20:35.086627] using decoder[VanillaChunkDecoder] ;
[2016-12-20T13:20:35.118315] Bulk processing finished successfully, id='1', numberOfMessages='1';
[…]
A log with fewer lines stripped can be found here.
We showed it's possible to setup a secure line between syslog-ng and Elasticsearch using Search Guard. All communication, be it between cluster nodes or between syslog-ng and the latter is handled using SSL.
-- [1]: https://www.balabit.com/blog/logging-to-elasticsearch-made-simple-with-syslog-ng/ [2]: https://www.balabit.com/documents/pdf/syslog-ng-whitepaper-elasticsearch-kibana.pdf [3]: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-on-rhelcentos/ [SG-SSL-doc]: https://github.com/floragunncom/search-guard-ssl-docs [SG-SSL-cert]: https://github.com/floragunncom/search-guard-ssl-docs/blob/master/certificates.md [FGcertgen]: https://floragunn.com/tls-certificate-generator/ [LS]: https://elastic.co/products/logstash [ES]: https://elastic.co/products/elasticsearch [xpack]: https://elastic.co/products/x-pack [shield]: https://elastic.co/products/shield [SG]: https://floragunn.com/searchguard/ [FGmotto]: https://floragunn.com/searchguard/searchguard-license-support/ [syslog_ng-sg]: syslog-ng/syslog-ng#1223 [SG-release]: https://github.com/floragunncom/search-guard-ssl/releases/tag/v2.4.1.16 [ES-release]: https://www.elastic.co/downloads/past-releases/elasticsearch-2-4-1 [SG5]: https://github.com/floragunncom/search-guard/tree/5.0.0 [SG-SSL]: https://floragunn.com/searchguard-ssl/ [syslog_ng-elastic-doc]: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose-guide-admin/html/reference-destination-elasticsearch2.html