Skip to content

Instantly share code, notes, and snippets.

@faxm0dem
Last active January 4, 2017 09:39
Show Gist options
  • Save faxm0dem/4c219bb5cd679092f86312b4e66af8b7 to your computer and use it in GitHub Desktop.
Save faxm0dem/4c219bb5cd679092f86312b4e66af8b7 to your computer and use it in GitHub Desktop.
syslog-ng searchguard blog

Securing your ESK stack for free using Search Guard

Introduction

Using syslog-ng to index events to your [Elasticsearch (ES)][ES] cluster has been possible natively for some time now ([see][1] [various][2] [blog][3] posts), but now with syslog-ng 3.9, you can even use Search Guard to encrypt and authenticate your connections to Elasticsearch.

The main reasons we're using syslog-ng over Logstash (LS) at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3) are:

  • superior performance,
  • lower system footprint, and
  • higher maintainability.

One of the reasons Elasticsearch is a popular choice to store and query logs is the fact that it's an opensource software (OSS) solution. Unfortunately, one of its drawbacks is security: the only official way to use flexible authorization policies and strong cryptography is to use [Shield][shield] (recently renamed to [X-pack][xpack]). And guess what, that's where the free part stops: Shield requires a full ES license, which can quickly empty your pockets, especially if your cluster grows.

Fortunately, the [Search Guard (SG)][SG] project implements strong security in the form of an ES plugin. Its core features are opensource, with additional functionality requiring a license. However, even the latter code is publicly available on GitHub. The license is free for non-commercial use, and much more reasonably priced than Shield for commercial usage: Search Guard is sponsored by its company: floragunn, true to its motto ["Scale your cluster, not your cost"][FGmotto].

That's where the subject of this post comes in: Search Guard support got [merged][syslog_ng-sg] to syslog-ng! Moreover, the Search Guard component required by syslog-ng doesn't require a commercial license, so you can use it for free. Hopefully after reading this post, you'll be able to fly your ESK stack to higher security standards.

Notes

This guide targets ES 2.4.1 (official [distribution from elastic.co][ES-release]), and [SG-ssl 2.4.1.16][SG-release] Versions for ES5 are [already available][SG5], and instructions shouldn't change much.

Search Guard

Search Guard started as a free plugin for ES 1.x, then was rewritten completely for ES 2.x, and recently gained support for ES 5.x (yes, elastic.co product's versioning is a total nightmare, and look! floragunn follows rapid change closely). Its main components are Search Guard and Search Guard SSL (both got merged since 5.x to ease deployment). We're not going into much detail about the former, as you'll be able to find a lot of info on its [homepage][SG-SSL] and [documentation][SG-SSL-doc]. Suffice it to say that its function is authentication and authorization, and that it supports many mechanisms like http simple, proxy, kerberos, etc. Also note that its developer team is very responsive, and very efficient at running after the rapid and unfortunate breaking of ES APIs by elastic.co. We'll discuss some more on the latter: Search Guard SSL is the plugin that handles encryption: it's the only one needed by syslog-ng, because it will talk natively with the other ES nodes on the encrypted transport layer (aka the 9300 tcp port).

Search Guard SSL

Citing the [project page][SG-SSL]:

Search Guard (®) SSL is a free and open source security plugin for Elasticsearch which provides SSL/TLS for the transport- and REST-layer.

Features

  • Node-to-node encryption through SSL/TLS (Transport layer)
  • Secure REST layer through HTTPS (SSL/TLS)
  • Supports JDK SSL and Open SSL
  • Works with Kibana 4, logstash and beats

We kindly invite the developers to add syslog-ng to that list as soon as this blog post is published ;-)

Configuration

In order to connect syslog-ng to your Search Guard SSL enabled ES cluster, you will need the following:

  • syslog-ng-3.9.1
  • a working search-guard-ssl enabled ES cluster
  • Elasticsearch installed on the syslog-ng node. (You need the same Elasticsearch version that the cluster uses, but it is not necessary to actually run it)
  • a copy of this blog post (or two)

For the impatient, the steps to follow are:

  1. Install the Search Guard SSL plugin on the syslog-ng node
  2. Generate a certificate for the syslog-ng node and add it to the cluster nodes' keystore
  3. Configure syslog-ng's elasticsearch2 destination to use client-mode(searchguard)
  4. Test
  5. Enjoy

Install search-guard plugin

sudo /usr/share/elasticsearch/bin/plugin install -b com.floragunn/search-guard-ssl/2.4.1.16

Certificates

Managing SG certificates is beyond the scope of this article. Please refer to the SG-SSL documentation regarding [certificates][SG-SSL-cert] for more details. The logical steps are the following:

  1. Generate a new key for your syslog-ng node
  2. Generate a CSR and send it to your CA (or self)
  3. CA sends back signed certificate
  4. Add the latter to the keystore of the node (SYSLOG_NG-NODE_NAME-keystore.jks)

For a quick'n'dirty PoC you can use floragunn's [certificate generator][FGcertgen] to get started in a breeze.

syslog-ng configuration

The configuration is pretty straightforward, just follow the [destination's documentation][syslog_ng-elastic-doc] using the correct client-mode.

In the examples, 192.168.42.42 is the syslog-server, and 192.168.42.69 one of the elasticsearch cluster nodes.

The following example will push messages to a daily index with a fixed type:

File: /etc/syslog-ng/syslog-ng.conf

destination d_elasticsearch {
  elasticsearch2(
    client-lib-dir("/usr/share/elasticsearch/plugins/search-guard-ssl/*.jar:/usr/share/elasticsearch/lib")
    index("syslog-${YEAR}.${MONTH}.${DAY}")
    type("syslog")
    time-zone("UTC")
    client_mode("searchguard")
    resource("/etc/syslog-ng/elasticsearch.yml")
  );
};

File: /etc/syslog-ng/elasticsearch.yml

cluster:
  name: elasticsearch
discovery:
  zen:
    ping:
      unicast:
        hosts:
          - 192.168.42.42
          - 192.168.42.69
node:
  name: syslog_ng_secure
  data; false
  master: false
path:
  home: /etc/syslog-ng
  conf: /etc/syslog-ng 
searchguard:
  ssl:
    transport:
      keystore_filepath: syslog_ng-keystore.jks
      keystore_password: changeit
      truststore_filepath: truststore.jks
      truststore_password: changeit
      enforce_hostname_verification: true

The key- and trust-stores should be placed into the directory path.conf = /etc/syslog-ng.

Note

The thorough reader might have frowned at the value of client-lib-dir(). We invite them to read the contents of github issue/1274 for more information on that matter. The unthorough reader may silently ignore this note.

The big picture


                               __--__--__--__--_____
                              (                     )
                             ( Elasticsearch Cluster )
                              (                     )
                             (                     )
                              (     es-node-1       )
                             (    192.168.42.69    )
+--------------+              (        ↑           )
|  syslog-ng   |             (         |            )
|192.168.42.42 | ---(sg)-->   (      (sg)       )
+--------------+             (         |          )
                              (        ↓         )
                             (      es-node-2     )
  __                          (   192.168.42.79    )
 ( o>                          (                  )
 ///\                         (                    )
 \V_/_                          (__---__-__--_--__)

Testing

The first thing to do is to test the syntax of our syslog-ng config using syslog-ng -f /etc/syslog-ng/syslog-ng.conf -s. The easiest way to test the config is then to run syslog-ng in the foreground using the command-line switch -F, and to activate the Full Monty debugging using the switches -d and -v: syslog-ng -f /etc/syslog-ng/syslog-ng.conf -Fdv.

If everything goes as expected, you should see something like the following:

[2016-12-20T13:19:08.289217] Module loaded and initialized successfully; module='mod-java'
[2016-12-20T13:19:08.303180] Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='mod-java.so', module='mod-java'
[2016-12-20T13:19:08.303201] Registering candidate plugin; module='mod-java', context='destination', name='java', preference='0'
[2016-12-20T13:19:08.452898] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/java-modules.jar;
[2016-12-20T13:19:08.453308] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/syslog-ng-core.jar;
[2016-12-20T13:19:08.453466] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/elastic-v2.jar;
[2016-12-20T13:19:08.455703] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/search-guard-ssl-2.4.1.16.jar;
[2016-12-20T13:19:08.738063] [syslog_ng] modules [], plugins [search-guard-ssl], sites [];
[…]
[2016-12-20T13:19:09.997536] Worker thread started; driver='d_elasticsearch#0'
[2016-12-20T13:19:09.997870] Checking cluster state..., cluster_name='elasticsearch';
[2016-12-20T13:19:10.043771] [id: 0xb6975e55, /192.168.42.42:55330 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.052281] Cluster is ready to work, cluster_name='elasticsearch';
[…now that everything's set up logs start to come in…]
[2016-12-20T13:20:34.548044] Incoming log entry; line='<27>Dec 20 13:20:34 root: /etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1'
[…and get written to elasticsearch…]
[2016-12-20T13:20:34.550464] Outgoing message; message='2016-12-20T13:20:34+01:00 root err /etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1\x0a'
[2016-12-20T13:20:34.550878] Outgoing log entry, json='{"PROGRAM":"root","PRIORITY":"err","PID":"11636","MESSAGE":"/etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1","ISODATE":"2016-12-20T12:20:34+00:00","HOST":"ccswissrp","FACILITY":"daemon","@timestamp":"2016-12-20T12:20:34+00:00"}';
[2016-12-20T13:20:35.078198] Start bulk processing, id='1';
[2016-12-20T13:20:35.086627] using decoder[VanillaChunkDecoder] ;
[2016-12-20T13:20:35.118315] Bulk processing finished successfully, id='1', numberOfMessages='1';
[…]

A log with fewer lines stripped can be found here.

Conclusion

We showed it's possible to setup a secure line between syslog-ng and Elasticsearch using Search Guard. All communication, be it between cluster nodes or between syslog-ng and the latter is handled using SSL.

-- [1]: https://www.balabit.com/blog/logging-to-elasticsearch-made-simple-with-syslog-ng/ [2]: https://www.balabit.com/documents/pdf/syslog-ng-whitepaper-elasticsearch-kibana.pdf [3]: https://www.balabit.com/blog/syslog-ng-and-elasticsearch-5-getting-started-on-rhelcentos/ [SG-SSL-doc]: https://github.com/floragunncom/search-guard-ssl-docs [SG-SSL-cert]: https://github.com/floragunncom/search-guard-ssl-docs/blob/master/certificates.md [FGcertgen]: https://floragunn.com/tls-certificate-generator/ [LS]: https://elastic.co/products/logstash [ES]: https://elastic.co/products/elasticsearch [xpack]: https://elastic.co/products/x-pack [shield]: https://elastic.co/products/shield [SG]: https://floragunn.com/searchguard/ [FGmotto]: https://floragunn.com/searchguard/searchguard-license-support/ [syslog_ng-sg]: syslog-ng/syslog-ng#1223 [SG-release]: https://github.com/floragunncom/search-guard-ssl/releases/tag/v2.4.1.16 [ES-release]: https://www.elastic.co/downloads/past-releases/elasticsearch-2-4-1 [SG5]: https://github.com/floragunncom/search-guard/tree/5.0.0 [SG-SSL]: https://floragunn.com/searchguard-ssl/ [syslog_ng-elastic-doc]: https://www.balabit.com/documents/syslog-ng-ose-3.8-guides/en/syslog-ng-ose-guide-admin/html/reference-destination-elasticsearch2.html

[2016-12-20T13:19:08.289217] Module loaded and initialized successfully; module='mod-java'
[2016-12-20T13:19:08.289323] Starting to read include file; filename='/usr/local/etc/scl.conf', depth='1'
[...]
[2016-12-20T13:19:08.289631] Adding include file; filename='/usr/local/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2'
[...]
[2016-12-20T13:19:08.289779] Reading path for candidate modules; path='/usr/local/lib/syslog-ng'
[...]
[2016-12-20T13:19:08.289847] Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='afsmtp.so', module='afsmtp'
[...]
[2016-12-20T13:19:08.303180] Reading shared object for a candidate module; path='/usr/local/lib/syslog-ng', fname='mod-java.so', module='mod-java'
[2016-12-20T13:19:08.303201] Registering candidate plugin; module='mod-java', context='destination', name='java', preference='0'
[...]
[2016-12-20T13:19:08.306828] Starting to read include file; filename='/usr/local/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2'
[2016-12-20T13:19:08.307027] Finishing include; filename='/usr/local/share/syslog-ng/include/scl/elasticsearch/plugin.conf', depth='2'
[...]
[2016-12-20T13:19:08.310249] Finishing include; content='destination block elasticsearch2', depth='1'
[...]
[2016-12-20T13:19:08.452898] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/java-modules.jar;
[2016-12-20T13:19:08.453308] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/syslog-ng-core.jar;
[2016-12-20T13:19:08.453466] Add path to classpath: /usr/local/lib/syslog-ng/java-modules/elastic-v2.jar;
[...]
[2016-12-20T13:19:08.455096] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-handler-4.0.37.Final.jar;
[2016-12-20T13:19:08.455237] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-buffer-4.0.37.Final.jar;
[2016-12-20T13:19:08.455368] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-common-4.0.37.Final.jar;
[2016-12-20T13:19:08.455483] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-codec-4.0.37.Final.jar;
[2016-12-20T13:19:08.455595] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/netty-transport-4.0.37.Final.jar;
[2016-12-20T13:19:08.455703] Add path to classpath: /usr/share/elasticsearch/plugins/search-guard-ssl/search-guard-ssl-2.4.1.16.jar;
[2016-12-20T13:19:08.455808] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queryparser-5.5.2.jar;
[2016-12-20T13:19:08.455900] Add path to classpath: /usr/share/elasticsearch/lib/lucene-join-5.5.2.jar;
[2016-12-20T13:19:08.456003] Add path to classpath: /usr/share/elasticsearch/lib/guava-18.0.jar;
[2016-12-20T13:19:08.456089] Add path to classpath: /usr/share/elasticsearch/lib/compress-lzf-1.0.2.jar;
[2016-12-20T13:19:08.456187] Add path to classpath: /usr/share/elasticsearch/lib/lucene-queries-5.5.2.jar;
[2016-12-20T13:19:08.456276] Add path to classpath: /usr/share/elasticsearch/lib/lucene-memory-5.5.2.jar;
[2016-12-20T13:19:08.456368] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial-5.5.2.jar;
[2016-12-20T13:19:08.456467] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-yaml-2.8.1.jar;
[2016-12-20T13:19:08.456572] Add path to classpath: /usr/share/elasticsearch/lib/jna-4.1.0.jar;
[2016-12-20T13:19:08.456658] Add path to classpath: /usr/share/elasticsearch/lib/apache-log4j-extras-1.2.17.jar;
[2016-12-20T13:19:08.456797] Add path to classpath: /usr/share/elasticsearch/lib/log4j-1.2.17.jar;
[2016-12-20T13:19:08.456895] Add path to classpath: /usr/share/elasticsearch/lib/jackson-core-2.8.1.jar;
[2016-12-20T13:19:08.457018] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-cbor-2.8.1.jar;
[2016-12-20T13:19:08.457123] Add path to classpath: /usr/share/elasticsearch/lib/lucene-core-5.5.2.jar;
[2016-12-20T13:19:08.457219] Add path to classpath: /usr/share/elasticsearch/lib/jsr166e-1.1.0.jar;
[2016-12-20T13:19:08.457315] Add path to classpath: /usr/share/elasticsearch/lib/lucene-analyzers-common-5.5.2.jar;
[2016-12-20T13:19:08.457426] Add path to classpath: /usr/share/elasticsearch/lib/jackson-dataformat-smile-2.8.1.jar;
[2016-12-20T13:19:08.457530] Add path to classpath: /usr/share/elasticsearch/lib/lucene-spatial3d-5.5.2.jar;
[2016-12-20T13:19:08.457622] Add path to classpath: /usr/share/elasticsearch/lib/hppc-0.7.1.jar;
[2016-12-20T13:19:08.457716] Add path to classpath: /usr/share/elasticsearch/lib/lucene-suggest-5.5.2.jar;
[2016-12-20T13:19:08.457809] Add path to classpath: /usr/share/elasticsearch/lib/lucene-highlighter-5.5.2.jar;
[2016-12-20T13:19:08.457911] Add path to classpath: /usr/share/elasticsearch/lib/lucene-misc-5.5.2.jar;
[2016-12-20T13:19:08.458017] Add path to classpath: /usr/share/elasticsearch/lib/spatial4j-0.5.jar;
[2016-12-20T13:19:08.458129] Add path to classpath: /usr/share/elasticsearch/lib/commons-cli-1.3.1.jar;
[2016-12-20T13:19:08.458226] Add path to classpath: /usr/share/elasticsearch/lib/netty-3.10.6.Final.jar;
[2016-12-20T13:19:08.458315] Add path to classpath: /usr/share/elasticsearch/lib/elasticsearch-2.4.1.jar;
[2016-12-20T13:19:08.458402] Add path to classpath: /usr/share/elasticsearch/lib/lucene-backward-codecs-5.5.2.jar;
[2016-12-20T13:19:08.458497] Add path to classpath: /usr/share/elasticsearch/lib/HdrHistogram-2.1.6.jar;
[2016-12-20T13:19:08.458588] Add path to classpath: /usr/share/elasticsearch/lib/snakeyaml-1.15.jar;
[2016-12-20T13:19:08.458730] Add path to classpath: /usr/share/elasticsearch/lib/lucene-grouping-5.5.2.jar;
[2016-12-20T13:19:08.458829] Add path to classpath: /usr/share/elasticsearch/lib/compiler-0.8.13.jar;
[2016-12-20T13:19:08.458924] Add path to classpath: /usr/share/elasticsearch/lib/t-digest-3.0.jar;
[2016-12-20T13:19:08.459270] Add path to classpath: /usr/share/elasticsearch/lib/securesm-1.0.jar;
[2016-12-20T13:19:08.459362] Add path to classpath: /usr/share/elasticsearch/lib/lucene-sandbox-5.5.2.jar;
[2016-12-20T13:19:08.459459] Add path to classpath: /usr/share/elasticsearch/lib/joda-convert-1.2.jar;
[2016-12-20T13:19:08.459552] Add path to classpath: /usr/share/elasticsearch/lib/jts-1.13.jar;
[2016-12-20T13:19:08.459638] Add path to classpath: /usr/share/elasticsearch/lib/joda-time-2.9.4.jar;
[...the following message just means we use search-guard-ssl without searchguard...]
[2016-12-20T13:19:08.736494] Search Guard 2 plugin not available;
[2016-12-20T13:19:08.738063] [syslog_ng] modules [], plugins [search-guard-ssl], sites [];
[2016-12-20T13:19:08.751529] [syslog_ng] creating thread_pool [force_merge], type [fixed], size [1], queue_size [null];
[2016-12-20T13:19:08.757625] [syslog_ng] creating thread_pool [percolate], type [fixed], size [4], queue_size [1k];
[2016-12-20T13:19:08.775043] [syslog_ng] creating thread_pool [fetch_shard_started], type [scaling], min [1], size [8], keep_alive [5m];
[2016-12-20T13:19:08.776080] [syslog_ng] creating thread_pool [listener], type [fixed], size [2], queue_size [null];
[2016-12-20T13:19:08.776276] [syslog_ng] creating thread_pool [index], type [fixed], size [4], queue_size [200];
[2016-12-20T13:19:08.776426] [syslog_ng] creating thread_pool [refresh], type [scaling], min [1], size [2], keep_alive[5m];
[2016-12-20T13:19:08.776612] [syslog_ng] creating thread_pool [suggest], type [fixed], size [4], queue_size [1k];
[2016-12-20T13:19:08.776722] [syslog_ng] creating thread_pool [generic], type [cached], keep_alive [30s];
[2016-12-20T13:19:08.777413] [syslog_ng] creating thread_pool [warmer], type [scaling], min [1], size [2], keep_alive [5m];
[2016-12-20T13:19:08.777582] [syslog_ng] creating thread_pool [search], type [fixed], size [7], queue_size [1k];
[2016-12-20T13:19:08.777701] [syslog_ng] creating thread_pool [flush], type [scaling], min [1], size [2], keep_alive [5m];
[2016-12-20T13:19:08.777825] [syslog_ng] creating thread_pool [fetch_shard_store], type [scaling], min [1], size [8], keep_alive [5m];
[2016-12-20T13:19:08.777955] [syslog_ng] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m];
[2016-12-20T13:19:08.778108] [syslog_ng] creating thread_pool [get], type [fixed], size [4], queue_size [1k];
[2016-12-20T13:19:08.778255] [syslog_ng] creating thread_pool [bulk], type [fixed], size [4], queue_size [50];
[2016-12-20T13:19:08.778381] [syslog_ng] creating thread_pool [snapshot], type [scaling], min [1], size [2], keep_alive [5m];
[2016-12-20T13:19:08.792761] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: org.apache.tomcat.jni.SSL;
[2016-12-20T13:19:08.993315] Config directory is /etc/syslog-ng/, from there the key- and truststore files are resolved relatively;
[2016-12-20T13:19:08.998971] Alias syslog_ng: is a certificate entry?false/is a key entry?true;
[2016-12-20T13:19:08.999018] No alias given, will trust all of the certificates in the store;
[2016-12-20T13:19:09.002163] Alias root-ca-chain: is a certificate entry?true/is a key entry?false;
[2016-12-20T13:19:09.002197] No alias given, will trust all of the certificates in the store;
[2016-12-20T13:19:09.028550] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256];
[2016-12-20T13:19:09.028624] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256];
[2016-12-20T13:19:09.028795] sslHTTPProvider:null with ciphers [];
[2016-12-20T13:19:09.028867] sslTransport protocols [TLSv1.2, TLSv1.1];
[2016-12-20T13:19:09.028908] sslHTTP protocols [TLSv1.2, TLSv1.1];
[2016-12-20T13:19:09.190969] [syslog_ng] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl];
[2016-12-20T13:19:09.418780] configuration:\x0a\x0alo\x0a inet 192.168.42.42 netmask:255.0.0.0 scope:host\x0a inet6 ::1 prefixlen:128 scope:host\x0a UP LOOPBACK mtu:65536 index:1\x0a\x0aeth0\x0a inet6 fe80::e2db:55ff:fee1:1c51 prefixlen:64 scope:link\x0a hardware E0:DB:55:E1:1C:51\x0a UP MULTICAST mtu:1500 index:2\x0a\x0awlan0\x0a inet 8.8.8.8 netmask:255.255.254.0 broadcast:8.8.8.8\x0a inet6 fe80::2677:3ff:fe96:2c54 prefixlen:64 scope:link\x0a hardware 24:11:11:11:11:54\x0a UP MULTICAST mtu:1500 index:4\x0a;
[2016-12-20T13:19:09.438145] using gathering [true];
[2016-12-20T13:19:09.458963] [syslog_ng] node_sampler_interval[5s];
[2016-12-20T13:19:09.473659] Using select timeout of 500;
[2016-12-20T13:19:09.473704] Epoll-bug workaround enabled = false;
[2016-12-20T13:19:09.509682] [syslog_ng] Node client configured for SSL;
[2016-12-20T13:19:09.515269] [syslog_ng] adding address [{#transport#-1}{192.168.42.42}{192.168.42.42:9300}];
[2016-12-20T13:19:09.616097] [syslog_ng] connected to node [{#transport#-1}{192.168.42.42}{192.168.42.42:9300}];
[2016-12-20T13:19:09.735809] [id: 0xfb9f2d84, /192.168.42.42:55304 => localhost/192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:09.915552] [id: 0x4c2b8911, /192.168.42.42:55306 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:09.918041] [id: 0x10b6f930, /192.168.42.42:55310 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:09.947804] [syslog_ng] connected to node [{esnode}{U3jAM6niSGejShYp0RWkjw}{192.168.42.42}{192.168.42.42:9300}];
[2016-12-20T13:19:09.949955] [id: 0x7f84b332, /192.168.42.42:55312 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:09.969665] [syslog_ng] address [192.168.42.42:9300] already exists with [{#transport#-1}{192.168.42.42}{192.168.42.42:9300}], ignoring...;
[2016-12-20T13:19:09.969752] Using transport based client mode with bulk message processing (flush_limit > 1) can cause high message dropping rate in case of connection broken, using node client mode is suggested;
[2016-12-20T13:19:09.970444] Running application hooks; hook='1'
[2016-12-20T13:19:09.970454] Running application hooks; hook='3'
[2016-12-20T13:19:09.970480] syslog-ng starting up; version='3.9.1'
[2016-12-20T13:19:09.997536] Worker thread started; driver='d_elasticsearch#0'
[2016-12-20T13:19:09.997870] Checking cluster state..., cluster_name='elasticsearch';
[2016-12-20T13:19:10.009329] [id: 0xb475d179, /192.168.42.42:55320 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.012056] [id: 0x02dc9e81, /192.168.42.42:55318 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.029239] [id: 0x103b2756, /192.168.42.42:55316 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.043023] [id: 0xa7fe53e9, /192.168.42.42:55314 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.043771] [id: 0xb6975e55, /192.168.42.42:55330 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.052281] Cluster is ready to work, cluster_name='elasticsearch';
[2016-12-20T13:19:10.057957] [id: 0x26966616, /192.168.42.42:55322 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.063407] [id: 0xf7790a9f, /192.168.42.42:55308 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.064288] [id: 0x19a69e95, /192.168.42.42:55324 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.064540] [id: 0x41bed5c2, /192.168.42.42:55326 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[2016-12-20T13:19:10.074107] [id: 0x2007fa07, /192.168.42.42:55328 => /192.168.42.42:9300] HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384;
[...now that everything's set up logs start to come in...]
[2016-12-20T13:20:34.548044] Incoming log entry; line='<27>Dec 20 13:20:34 root: /etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1'
[...and get written to elasticsearch...]
[2016-12-20T13:20:34.550464] Outgoing message; message='2016-12-20T13:20:34+01:00 root err /etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1\x0a'
[2016-12-20T13:20:34.550878] Outgoing log entry, json='{"PROGRAM":"root","PRIORITY":"err","PID":"11636","MESSAGE":"/etc/dhcp/dhclient-enter-hooks.d/avahi-autoipd returned non-zero exit status 1","ISODATE":"2016-12-20T12:20:34+00:00","HOST":"ccswissrp","FACILITY":"daemon","@timestamp":"2016-12-20T12:20:34+00:00"}';
[2016-12-20T13:20:35.078198] Start bulk processing, id='1';
[2016-12-20T13:20:35.086627] using decoder[VanillaChunkDecoder] ;
[2016-12-20T13:20:35.118315] Bulk processing finished successfully, id='1', numberOfMessages='1';
[...]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment