Last active
August 29, 2015 14:04
-
-
Save neilmillard/9c9f792154d90c843cca to your computer and use it in GitHub Desktop.
logstash forwarder install
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<h2>Set Up Logstash Forwarder</h2> | |
<p><strong>Note</strong>: Do these steps for each server that you want to send logs to your Logstash Server. For instructions on installing Logstash Forwarder on Debian-based Linux distributions (e.g. Ubuntu, Debian, etc.), refer to the <a href="https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04#SetUpLogstashForwarder">Set Up Logstash Forwarder section of the Ubuntu variation of this tutorial</a>.</p> | |
<h3>Copy SSL Certificate and Logstash Forwarder Package</h3> | |
<p>On <strong>Logstash Server</strong>, copy the SSL certificate to <strong>Server</strong> (substitute with your own login):</p> | |
<pre><code>scp /etc/pki/tls/certs/logstash-forwarder.crt <span class="highlight">user</span>@<span class="highlight">server_private_IP</span>:/tmp | |
</code></pre> | |
<h3>Install Logstash Forwarder Package</h3> | |
<p>On <strong>Server</strong>, download the Logstash Forwarder RPM to your home directory:</p> | |
<pre><code>cd ~; curl -O http://packages.elasticsearch.org/logstashforwarder/centos/logstash-forwarder-0.3.1-1.x86_64.rpm | |
</code></pre> | |
<p>Then install the Logstash Forwarder Package:</p> | |
<pre><code>sudo rpm -ivh ~/logstash-forwarder-0.3.1-1.x86_64.rpm | |
</code></pre> | |
<p>Next, you will want to install the Logstash Forwarder init script, so it starts on bootup. We will use the init script provided by logstashbook.com:</p> | |
<pre><code>cd /etc/init.d/; sudo curl -o logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_init | |
sudo chmod +x logstash-forwarder | |
</code></pre> | |
<p>The init script depends on a file called <code>/etc/sysconfig/logstash-forwarder</code>. A sample file is available to download:</p> | |
<pre><code>sudo curl -o /etc/sysconfig/logstash-forwarder http://logstashbook.com/code/4/logstash_forwarder_redhat_sysconfig | |
</code></pre> | |
<p>Open it for editing:</p> | |
<pre><code>sudo vi /etc/sysconfig/logstash-forwarder | |
</code></pre> | |
<p>And modify the <code>LOGSTASH_FORWARDER_OPTIONS</code> value so it looks like the following:</p> | |
<pre><code>LOGSTASH_FORWARDER_OPTIONS="-config /etc/logstash-forwarder -spool-size 100" | |
</code></pre> | |
<p>Save and quit.</p> | |
<p>Now copy the SSL certificate into the appropriate location (/etc/pki/tls/certs):</p> | |
<pre><code>sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/ | |
</code></pre> | |
<h3>Configure Logstash Forwarder</h3> | |
<p>On <strong>Server</strong>, create and edit Logstash Forwarder configuration file, which is in JSON format:</p> | |
<pre><code>sudo vi /etc/logstash-forwarder | |
</code></pre> | |
<p>Now add the following lines into the file, substituting in your Logstash Server's private IP address for <code>logstash_server_private_IP</code>:</p> | |
<pre><code>{ | |
"network": { | |
"servers": [ "<span class="highlight">logstash_server_private_IP</span>:5000" ], | |
"timeout": 15, | |
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt" | |
}, | |
"files": [ | |
{ | |
"paths": [ | |
"/var/log/messages", | |
"/var/log/secure" | |
], | |
"fields": { "type": "syslog" } | |
} | |
] | |
} | |
</code></pre> | |
<p>Save and quit. This configures Logstash Forwarder to connect to your Logstash Server on port 5000 (the port that we specified an input for earlier), and uses the SSL certificate that we created earlier. The <em>paths</em> section specifies which log files to send (here we specify <em>messages</em> and <em>secure</em>), and the <em>type</em> section specifies that these logs are of type "syslog* (which is the type that our filter is looking for).</p> | |
<p>Note that this is where you would add more files/types to configure Logstash Forwarder to other log files to Logstash on port 5000.</p> | |
<p>Now we will want to add the Logstash Forwarder service with <em>chkconfig</em>:</p> | |
<pre><code>sudo chkconfig --add logstash-forwarder | |
</code></pre> | |
<p>Now start Logstash Forwarder to put our changes into place:</p> | |
<pre><code>sudo service logstash-forwarder start | |
</code></pre> | |
<p>Now Logstash Forwarder is sending <em>messages</em> and <em>auth.log</em> to your Logstash Server! Repeat this process for all of the other servers that you wish to gather logs for.</p> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"network": { | |
"servers": [ "kibana.ipaddr:5000" ], | |
"timeout": 15, | |
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt" | |
}, | |
"files": [ | |
{ | |
"paths": [ | |
"/var/log/messages", | |
"/var/log/secure" | |
], | |
"fields": { "type": "syslog" } | |
} | |
] | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#! /bin/sh | |
# Modified Logstash-forwarder startup script | |
# | |
# chkconfig: 345 99 99 | |
# description: logstash-forwarder | |
# processname: logstash-forwarder | |
DESC="logstash shipper" | |
NAME=logstash-forwarder | |
SCRIPT=/etc/init.d/$NAME | |
STOP_TIMEOUT=5 | |
FORWARDER_BIN="/opt/logstash-forwarder/bin/logstash-forwarder" | |
pidfile=/var/run/$NAME.pid | |
# Source function library | |
. /etc/rc.d/init.d/functions | |
if [ -f /etc/sysconfig/$NAME ]; then | |
. /etc/sysconfig/$NAME | |
fi | |
write_pid () { | |
PIDTEMP=`pgrep -f ${FORWARDER_BIN}` | |
# PID not found | |
if [ "x$PIDTEMP" = "x" ]; then | |
echo -1 > $pidfile | |
else | |
echo $PIDTEMP > $pidfile | |
fi | |
} | |
get_pid() { | |
return `pgrep -f ${FORWARDER_BIN}` | |
} | |
start() { | |
echo -n $"Starting $NAME: " | |
nohup ${FORWARDER_BIN} ${OPTIONS} > /var/log/$NAME/$NAME.log 2>&1 & | |
write_pid | |
PID=get_pid | |
[ $PID = -1 ] && failure || success | |
echo | |
} | |
stop () { | |
echo -n $"Stopping $NAME: " | |
killproc -p ${pidfile} -d ${STOP_TIMEOUT} ${FORWARDER_BIN} | |
RETVAL=$? | |
[ $RETVAL = 0 ] && rm -f ${pidfile} && success || failure | |
echo | |
} | |
case $1 in | |
start) | |
start | |
;; | |
stop) | |
stop | |
exit 0; | |
;; | |
restart) | |
stop | |
start | |
;; | |
status) | |
status -p ${pidfile} ${FORWARDER_BIN} | |
;; | |
*) | |
echo $"Usage: $0 {start|stop|restart|status}" | |
RETVAL=1 | |
esac | |
exit 0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# From The Logstash Book | |
# The original of this file can be found at: http://logstashbook.com/code/index.html | |
# | |
# Options for the Logstash Forwarder | |
#LOGSTASH_FORWARDER_OPTIONS="-config /etc/logstash-forwarder/logstash-forwarder.conf" | |
OPTIONS="-config /etc/logstash-forwarder -spool-size 100" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment