Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Installing logstash server and client

Installing logstash

Server install

Install

Install the Java prerequisite:

sudo aptitude install openjdk-7-jre-headless

Set up the package repository so that we can use aptitude (docs):

wget -qO - https://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elasticsearch.org/logstash/1.5/debian stable main" | sudo tee -a /etc/apt/sources.list

Install logstash itself:

sudo aptitude install logstash

Generate SSL certificates

Docs: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04

Create the needed directories:

sudo mkdir -p /etc/pki/tls/{certs,private}

Create FQDN (DNS) based certificates (change CN as appropriate):

cd /etc/pki/tls; sudo openssl req -subj '/CN=*.bwbaugh.com/' -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

Configure

Lumberjack input

Choose a port.

  • Replace the 5000 example below.
  • Add the port to the inbound rules for the security group.

Create and edit a configuration file for the logstash-forwarder:

sudo vim /etc/logstash/conf.d/01-lumberjack-input.conf

with the following contents:

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

Syslog

Create the file:

sudo vim /etc/logstash/conf.d/10-syslog.conf

with the following contents:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

Debugging output

Create:

sudo vim /etc/logstash/conf.d/30-output-stdout.conf

with:

output {
  stdout { codec => json }
}

S3 output

Create:

sudo vim /etc/logstash/conf.d/30-syslog-output.conf

with:

output {
  if [type] == "syslog" {
    s3 {
      region => "us-west-2"
      bucket => "YOUR_BUCKET_NAME_HERE"
      canned_acl => "private"
  
      codec => "json_lines"
  
      size_file => 10485760
      time_file => 60

      prefix => "syslog/"
    }
  }
}

Start it

Start or restart logstash:

sudo service logstash restart

Logs

Check /var/log/logstash/ for the daemon's output

Client install

Install

Set up the package repository:

wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list

Install logstash-forwarder itself:

sudo aptitude update
sudo aptitude install logstash-forwarder

Set up SSL certificates

Copy the Logstash server's SSL certificate:

sudo mkdir -p /etc/pki/tls/certs
# Either copy/paste or use `scp`.
sudo vim /etc/pki/tls/certs/logstash-forwarder.crt

Configure

Edit the configuration file:

sudo vim /etc/logstash-forwarder.conf

Under the network section add the following with the proper host and port:

    "servers": [ "logstash_server_private_address:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"

Under the files section add:

    {
      "paths": [
        "/var/log/syslog",
        "/var/log/auth.log"
       ],
      "fields": { "type": "syslog" }
    }

Start it

Start or restart logstash-forwarder:

sudo service logstash-forwarder restart

Useful tips

The logstash-forwarder keeps track of files it has already sent in:

/var/lib/logstash-forwarder/.logstash-forwarder
@spemble

This comment has been minimized.

Copy link

@spemble spemble commented Sep 15, 2015

I'm having trouble getting this to work - I think it may be due to new versions of the packages. The package repository cannot find any package named or described as "logstash-forwarder".

@kpcool

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.