logstash forwarder install

readme.html

<h2>Set Up Logstash Forwarder</h2>

<p><strong>Note</strong>: Do these steps for each server that you want to send logs to your Logstash Server. For instructions on installing Logstash Forwarder on Debian-based Linux distributions (e.g. Ubuntu, Debian, etc.), refer to the <a href="https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04#SetUpLogstashForwarder">Set Up Logstash Forwarder section of the Ubuntu variation of this tutorial</a>.</p>

<h3>Copy SSL Certificate and Logstash Forwarder Package</h3>

<p>On <strong>Logstash Server</strong>, copy the SSL certificate to <strong>Server</strong> (substitute with your own login):</p>

<pre><code>scp /etc/pki/tls/certs/logstash-forwarder.crt <span class="highlight">user</span>@<span class="highlight">server_private_IP</span>:/tmp
</code></pre>

<h3>Install Logstash Forwarder Package</h3>

<p>On <strong>Server</strong>, download the Logstash Forwarder RPM to your home directory:</p>

<pre><code>cd ~; curl -O http://packages.elasticsearch.org/logstashforwarder/centos/logstash-forwarder-0.3.1-1.x86_64.rpm
</code></pre>

<p>Then install the Logstash Forwarder Package:</p>

<pre><code>sudo rpm -ivh ~/logstash-forwarder-0.3.1-1.x86_64.rpm
</code></pre>

<p>Next, you will want to install the Logstash Forwarder init script, so it starts on bootup. We will use the init script provided by logstashbook.com:</p>

<pre><code>cd /etc/init.d/; sudo curl -o logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_init
sudo chmod +x logstash-forwarder
</code></pre>

<p>The init script depends on a file called <code>/etc/sysconfig/logstash-forwarder</code>. A sample file is available to download:</p>

<pre><code>sudo curl -o /etc/sysconfig/logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_sysconfig
</code></pre>

<p>Open it for editing:</p>

<pre><code>sudo vi /etc/sysconfig/logstash-forwarder
</code></pre>

<p>And modify the <code>LOGSTASH_FORWARDER_OPTIONS</code> value so it looks like the following:</p>

<pre><code>LOGSTASH_FORWARDER_OPTIONS="-config /etc/logstash-forwarder -spool-size 100"
</code></pre>

<p>Save and quit.</p>

<p>Now copy the SSL certificate into the appropriate location (/etc/pki/tls/certs):</p>

<pre><code>sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/
</code></pre>

<h3>Configure Logstash Forwarder</h3>

<p>On <strong>Server</strong>, create and edit Logstash Forwarder configuration file, which is in JSON format:</p>

<pre><code>sudo vi /etc/logstash-forwarder
</code></pre>

<p>Now add the following lines into the file, substituting in your Logstash Server's private IP address for <code>logstash_server_private_IP</code>:</p>

<pre><code>{
  "network": {
    "servers": [ "<span class="highlight">logstash_server_private_IP</span>:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
       ],
      "fields": { "type": "syslog" }
    }
   ]
}
</code></pre>

<p>Save and quit. This configures Logstash Forwarder to connect to your Logstash Server on port 5000 (the port that we specified an input for earlier), and uses the SSL certificate that we created earlier. The <em>paths</em> section specifies which log files to send (here we specify <em>messages</em> and <em>secure</em>), and the <em>type</em> section specifies that these logs are of type "syslog* (which is the type that our filter is looking for).</p>

<p>Note that this is where you would add more files/types to configure Logstash Forwarder to other log files to Logstash on port 5000.</p>

<p>Now we will want to add the Logstash Forwarder service with <em>chkconfig</em>:</p>

<pre><code>sudo chkconfig --add logstash-forwarder
</code></pre>

<p>Now start Logstash Forwarder to put our changes into place:</p>

<pre><code>sudo service logstash-forwarder start
</code></pre>

<p>Now Logstash Forwarder is sending <em>messages</em> and <em>auth.log</em> to your Logstash Server! Repeat this process for all of the other servers that you wish to gather logs for.</p>

etc-logstash-forwarder

{
  "network": {
    "servers": [ "kibana.ipaddr:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
       ],
      "fields": { "type": "syslog" }
    }
   ]
}

logstash_init

#! /bin/sh
# Modified Logstash-forwarder startup script
#
# chkconfig: 345 99 99
# description: logstash-forwarder
# processname: logstash-forwarder

DESC="logstash shipper"
NAME=logstash-forwarder
SCRIPT=/etc/init.d/$NAME
STOP_TIMEOUT=5
FORWARDER_BIN="/opt/logstash-forwarder/bin/logstash-forwarder"
pidfile=/var/run/$NAME.pid

# Source function library
. /etc/rc.d/init.d/functions
if [ -f /etc/sysconfig/$NAME ]; then
        . /etc/sysconfig/$NAME
fi

write_pid () {
        PIDTEMP=`pgrep -f ${FORWARDER_BIN}`
        # PID not found
        if [ "x$PIDTEMP" = "x" ]; then
                echo -1 > $pidfile
        else
                echo $PIDTEMP > $pidfile
        fi
}

get_pid() {
        return `pgrep -f ${FORWARDER_BIN}`
}

start() {
        echo -n $"Starting $NAME:  "
        nohup ${FORWARDER_BIN} ${OPTIONS} > /var/log/$NAME/$NAME.log 2>&1 &
        write_pid
        PID=get_pid
        [ $PID = -1 ] && failure || success
        echo
}

stop () {
        echo -n $"Stopping $NAME:  "
        killproc -p ${pidfile} -d ${STOP_TIMEOUT} ${FORWARDER_BIN}
        RETVAL=$?
        [ $RETVAL = 0 ] && rm -f ${pidfile} && success || failure
        echo
}

case $1 in
start)
        start
        ;;

stop)
        stop
        exit 0;
        ;;

restart)
        stop
        start
        ;;

status)
        status -p ${pidfile} ${FORWARDER_BIN}
        ;;

*)
        echo $"Usage: $0 {start|stop|restart|status}"
        RETVAL=1

esac
exit 0

sysconfig-forwarder

# From The Logstash Book
# The original of this file can be found at: http://logstashbook.com/code/index.html
#
# Options for the Logstash Forwarder
#LOGSTASH_FORWARDER_OPTIONS="-config /etc/logstash-forwarder/logstash-forwarder.conf"
OPTIONS="-config /etc/logstash-forwarder -spool-size 100"