Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Installing logstash server and client

Installing logstash

Server install

Install

Install the Java prerequisite:

sudo aptitude install openjdk-7-jre-headless

Set up the package repository so that we can use aptitude (docs):

wget -qO - https://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elasticsearch.org/logstash/1.5/debian stable main" | sudo tee -a /etc/apt/sources.list

Install logstash itself:

sudo aptitude install logstash

Generate SSL certificates

Docs: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04

Create the needed directories:

sudo mkdir -p /etc/pki/tls/{certs,private}

Create FQDN (DNS) based certificates (change CN as appropriate):

cd /etc/pki/tls; sudo openssl req -subj '/CN=*.bwbaugh.com/' -x509 -days 3650 -batch -nodes -newkey rsa:4096 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

Configure

Lumberjack input

Choose a port.

  • Replace the 5000 example below.
  • Add the port to the inbound rules for the security group.

Create and edit a configuration file for the logstash-forwarder:

sudo vim /etc/logstash/conf.d/01-lumberjack-input.conf

with the following contents:

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

Syslog

Create the file:

sudo vim /etc/logstash/conf.d/10-syslog.conf

with the following contents:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

Debugging output

Create:

sudo vim /etc/logstash/conf.d/30-output-stdout.conf

with:

output {
  stdout { codec => json }
}

S3 output

Create:

sudo vim /etc/logstash/conf.d/30-syslog-output.conf

with:

output {
  if [type] == "syslog" {
    s3 {
      region => "us-west-2"
      bucket => "YOUR_BUCKET_NAME_HERE"
      canned_acl => "private"
  
      codec => "json_lines"
  
      size_file => 10485760
      time_file => 60

      prefix => "syslog/"
    }
  }
}

Start it

Start or restart logstash:

sudo service logstash restart

Logs

Check /var/log/logstash/ for the daemon's output

Client install

Install

Set up the package repository:

wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list

Install logstash-forwarder itself:

sudo aptitude update
sudo aptitude install logstash-forwarder

Set up SSL certificates

Copy the Logstash server's SSL certificate:

sudo mkdir -p /etc/pki/tls/certs
# Either copy/paste or use `scp`.
sudo vim /etc/pki/tls/certs/logstash-forwarder.crt

Configure

Edit the configuration file:

sudo vim /etc/logstash-forwarder.conf

Under the network section add the following with the proper host and port:

    "servers": [ "logstash_server_private_address:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"

Under the files section add:

    {
      "paths": [
        "/var/log/syslog",
        "/var/log/auth.log"
       ],
      "fields": { "type": "syslog" }
    }

Start it

Start or restart logstash-forwarder:

sudo service logstash-forwarder restart

Useful tips

The logstash-forwarder keeps track of files it has already sent in:

/var/lib/logstash-forwarder/.logstash-forwarder
@spemble
Copy link

spemble commented Sep 15, 2015

I'm having trouble getting this to work - I think it may be due to new versions of the packages. The package repository cannot find any package named or described as "logstash-forwarder".

@kpcool
Copy link

kpcool commented Dec 11, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment